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ABSTRACT 



This paper presents an alternative means of assessing 
faculty teaching that can be used by administrators. This method has been 
derived from in-depth faculty and student interviews, and its criterion of 
quality has been validated with empirical data through computer sensitivity 
analysis. The assessment process has been successfully tested in clinical 
teaching trials. Interviews with faculty have identified that their implicit 
expectations for professional development can be described in terms of 
technical skills, professional competence, and professional attitudes. This 
Three Ability Framework (3AF) is explained to the students by the lecturer, 
who is then rated by the students on each of these abilities. The lecturer 
rates himself or herself on the same abilities, and the alignment between the 
self-rating and the students' ratings is used as a measure of the 
effectiveness of teaching. The indicator of effective teaching is that the 
students and the instructor are working toward the same changes . This method 
promotes a positive teaching and learning culture in direct and indirect ways 
as it encourages teaching that promotes students' critical and evaluative 
thinking and high standards in technical skills and professional values. 
(Contains 33 references.) (SLD) 
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Many institutions now assess the quality of teaching by using anonymous student evaluations 
solicited towards the end of a course. They cannot feedback for the benefit of the students and 
the course to which they apply precisely because they are post-mortem evaluations. However, 
they are used by the institutions in a different context, for promotion and tenure decisions. This 
separation of context and purpose results in lecturers employing techniques of managing their 
teaching that deleteriously skew the institutional learning culture. For example, one technique 
of ensuring high student evaluations is to overgrade continuous in-course assessment and block- 
teach model examination answers. Students who have adopted a successful learning style for 
such courses are over-confident in their ability to transfer their learning to other courses and 
have diminished self-directed learning skills. The ripple-effect on the institutional learning culture 
is that lecturers on dependent courses must employ similar techniques. Dumbing down courses 
to increase popularity rating results in many problems for the institutional learning culture. One 
resulting problem is the difficulty these students then have in completing research theses and 
assignments. Research courses and research course components, which of necessity are dependent 
on transfer of learning, critical and evaluative thinking and self-directed learning, are being 
replaced by taught courses. 

This paper presents an alternative means of monitoring teaching quality that can be used 
by lecturers and administrators. It is designed to promote an alternative positive teaching and 
learning culture within the institution. This alternative method of assessment was derived from 
in-depth faculty and student interviews. Its criterion of quality has been validated on empirical 
data by computer sensitivity analysis and the assessment process successfully tested in clinical 
teaching trials. 

This assessment method explicitly promotes students’ understanding and professional 
attitudes, as well as their traditional skills, by operationally defining the assessment of skills, 
understanding and attitudes. It respects professional freedom and the inherent culture of each 
subject area by giving lecturers the responsibility of promoting the culture of their subjects 
through their teaching and assessment of its skills, understandings and attitudes. It allows faculty 
to adjust their in-course teaching and assessment schemes to the changing needs of the student 
body and minority students in their classes. The method results in a single administrative 
decision-point number that is a measure of quality teaching as it applies to (i) a particular 
individual, or (ii) a minority group or (iii) the whole class. 



Bastick, T. (1999, August). A technique for improving institutional learning culture by monitoring the 
qualityof teaching. Paper presented at “New Realities - Renewed Institutions” for the European Association 
for Institutional Research (EADR.) 2 1st Annual Forum, Lund, Sweden. 
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Negative influences on an institution’s teaching and learning culture 

Many organisational factors and resources affect the teaching and learning culture of an educational 
institution. This paper focuses on the use of assessing quality teaching as a means of developing a positive 
teaching and learning institutional culture. There is a business maxim that states “what gets measured gets 
done and what gets rewarded gets repeated” (Friend, 1972). In educational institutions what gets rewarded 
is (i) the assessment of faculty teaching and (ii) the assessment of student learning. It is how the assessment 
of how teaching is done and how the assessment of learning is done that has a great influence on the 
learning and teaching culture of the institution (Beichner, 1994; Hake, 1998; Halloun & Hestenes, 1985; 
Halloun, Hake, Mosca, & Hestenes, 1997; Hestenes, Wells, & Swackhamer 1992; Hestenes & Wells, 
1992; SokolofF & Thornton, 1997). These three aspects (i) teaching (ii) learning and (iii) the assessment of 
both, are so inter-related that they should be considered together. 

Problems with current assessments 

Unfortunately traditional forms of assessment lack integration and problems caused by these traditional 
forms of assessment negatively influence the learning and teaching culture of institutions. The common 
traditional assessment of faculty teaching is via the use of student opinionnaires. For example, Seldin 
(1984) found that administrators utilized student opinionnaires in two-thirds of 616 institutions surveyed. 
Student opinionnaires have been criticised as popularity contests, where staff reduce the standards of their 
courses and lead students to expect high grades in order to ‘win their vote’ (Greenwald, 1997; Greenwald 
& Gillmore, 1997; Howard & Maxwell, 1982; Marsh & Dunkin, 1997; Marsh & Roche, 1997, 1998). 
Faculty are expected to change their teaching in view of this anonymous feedback (Moses, 1996). Often 
these opinionnaires are anonymous because students fear retribution for ‘failing’ their teachers or might be 
thought to be soliciting favourable grades for favourable assessments of their teachers. Faculty feel 
uncomfortable in this one-sided contest where they can be anonymously failed by the students they teach, 
yet have little influence over the selection of the students they must teach. Some faculty feel that having to 
change their course emphasis in order to please naive students is an infringement of their professional 
freedom to teach how they believe, in their best professional judgement, their subject should be taught 
(Crumbley, 1995). It is questionable how much a university should be an academic leader or be managed 
as a business subservient to the student customer where ‘the customer is always right’. These influences 
have been contributing to a negative culture of low academic standards, demotivated professionalism and 
mutual distrust for some years (Arreola, 1983; Cashin, 1983; Cherry, Grant, & Kalinos, 1988). For example, 
“What is called development, growth, and self-improvement today becomes the means by which decisions 
for institutional personnel management purposes are made tomorrow. Faculties become wary and suspicious 
of this double message involved in the evaluation system” (Mark, 1982, p. 168). 

The traditional assessment of student learning is via examinations and coursework assignments of 
various kinds. Here the one-sided game is against the student who has little educational recourse and so 
can only resort to complaint about the course and the faculty - even to the extent of litigation. To protect 
themselves from the ‘court case student’ faculty favour objective assessment that does not expose their 
professional judgements about the students’ work. Such objective assessments tend to emphasise Bloom’s 
lowest cognitive level of rote learning. For example, “McKeachie (1987) has recently reminded educational 
researchers and practitioners that the achievement tests assessing student learning in the sorts of studies 
reviewed here typically measure lower-level educational objectives such as memory of facts and definitions 
rather than higher-level outcomes such as critical thinking and problem-solving that are usually taken as 
important in higher education.” Feldman (1989). Adult students in professional subjects rightly devalue 
courses that emphasise only rote learning, to the extent that they are prepared to cheat in order to maximise 
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their meaningless marks. Problem attendance is a feature of such courses - students get the lecture notes 
from those who do attend. Students learn not to criticise the views of faculty but to unquestionable do as 
they are told, and to parrot what they believe Faculty expects them to regurgitate in examinations. Such is 
the negative effect traditional assessments of teaching and learning have on the teaching and learning 
culture of the institution. Research has indicated that these problems are in part due to misunderstanding of 
mutual expectations (Bastick, 1995). Faculty lack clarity in explicitly stating their expectations and relating 
these to their teaching and assessment of the students. The students misunderstand what is expected of 
them and are confused. 

Positive influences on an institution’s teaching and learning culture 

What is a positive teaching and learning institutional culture? The literature indicates that is one that 
encourages staff and students to be independent critical thinkers developing the attitudes and values of 
their profession. One where students and faculty value and enjoy the work they are doing. One where 
faculty and students respectfully work together based on a foundation of mutual trust. However, if an 
alternative method of assessment is to promote these changes, then first, that method must accommodate 
the wide institutional variations that exist in assessment preferences and aim to improve teaching and 
learning by allowing those lecturers who use it to appreciate more fitting styles of teaching and enable 
them to allow their students to adopt more fitting styles of learning. Secondly, an alternative method of 
assessing teaching and learning must resolve the misunderstandings and confusions about mutual 
expectations in order to avoid the problems that lead to a negative teaching and learning culture. Thirdly, 
an alternative method must promote a positive teaching and learning culture by (i) ensuring students and 
faculty understand each other’s expectations and (ii) by ensuring that students and faculty are both working 
towards the same expectations (Abrami, d’Apollonia, & Cohen, 1990; Abrami, 1989; Bastick, 1995; Bortz, 
1984; L’Hommedieu, Menges, & Brinko, 1990; Miller, 1986; Scriven, 1994, 1995). 

It is also important to separate evaluations of attainment from evaluations of enjoyment, so that 
student evaluations of course enjoyment are not simply ‘smile sheets’ misused as assessments of academic 
attainment (Hake, in press). Hence, the two separate criteria of effective teaching used by this alternate 
method of assessment are to maximise (i) the academic attainment of the students and (ii) the students’ 
and the lecturer’s enjoyment of the course. The measurable indicator of effective teaching used is that the 
students and the lecturer are working towards the same expectations. The construct validity that this 
measurable indicator assesses the criteria is p<0.01 for both (i) and (ii) (Bastick, 1995) 

Interviews with faculty on professional courses have indicated that their implicit expectations can be 
described and assessed in terms of three abilities (i) technical skills - rote learning, assessed by the accuracy 
of reproduction (ii) professional competence - appropriate transfer of skills to a novel situation, assessed 
by the justification of appropriateness and (iii) professional attitudes - the integration of one’s life and 
work by one’s values and beliefs, assessed by demonstration (Bastick, 1995). Faculty can be assisted in 
making these expectations explicit and in designing coursework and examinations that offer opportunities 
for assessing these three abilities. This professional development can be expected to improve the quality of 
their teaching (Askew, Brown, Rhodes, Wiliam, & Johnson, 1997). It should be their professional prerogative 
to decide, and justify to their peers and their students, the emphasis they judge should be given to each of 
the three abilities on their courses. These judgements will depend on the subject, its level and the professional 
inclination of the lecturer. For example, lecturers on B.Ed courses expect an emphasis on technical skills 
in the first year, moving to an emphasis on professional competence in the second year and a greater 
emphasis on professional attitudes in the third year. 
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The Three Ability Framework (3AF) 

Three Ability Framework (3AF) is a complete alternative to post-mortem student opinionnaires and 
their attendant problems. The design of its management framework incorporates the positive influences 
mentioned above to enhance teaching/leaming culture to the advantage of the institution, faculty and 
students. One part of the management framework is the use of the 3 AF feedback form. The 3AF feedback 
form has only 6 necessary ratings that take less than 5 minutes to complete. Hence, the form can conveniently 
be used many times by the lecturer during the course for in-course tracking of teaching quality. At the end 
of the course it can be used by the administration to give a single decision point number representing the 
quality of teaching. This is also fairer and less threatening to faculty who, by previous uses of the form, 
have had opportunities to respond to the feedback and so improve the course for their students and for 
themselves. As mentioned above, previous research has connected teaching and assessment problems on 
professional courses with staff/student mis-matched expectations of three abilities. These three abilities 
are technical skills, professional competence and professional attitudes (Bastick, 1995). The 3AF uses the 
matching of stafif/student expectations on these three abilities as the basis of teaching effectiveness. 



The 3 AF form asks for two ratings of each of these abilities; ratings of how it is now on the course and 
ratings of how the student would want it to be. This is shown in figure 1 . The form takes longer to complete 
when additional information is requested. 



Estimate, for you personally, how much this course emphasises, and should emphasise (i) Skills, 
(ii) Competence and (iii) Attitudes? Do this for both how the course is now, and for how the 
course should be - write a number in each box. 

As it is now on this course As it should be on this course 



(i) Emphasis on Skills 



Your 
estimate 
out of 100 



Your 
estimate 
out of 100 



As it is now on this course As it should be on this course 



(ii) Emphasis on Competence 



Your 
estimate 
out of 1 00 



Your 
estimate 
out of 100 



As it is now on this course As it should be on this course 



(iii) Emphasis on Attitudes 



Your 
estimate 
out of 100 



Your 
estimate 
out of 1 00 



Figure 1 : 3AF feedback form asking for two ratings of each of the three abilities 



Technical Skills refer to the traditional speed and accuracy of reproducing facts and processes and is 
assessed by timed accuracy of reproduction. Professional competence refers to the ability to use the skills 
in a novel situation or extend these skills in a novel way. The assessment is by justification of the 
appropriateness of what is done. Professional attitudes refers to values that are appropriate to the subject. 
They are assessed by demonstration in practical situations. 
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From the difference in each pair of ratings it is possible to calculate each student’s expectation for 
change in that ability. The lecturer completes the same ratings at the same time, and from the lecturer’s 
form it is possible to also calculate, in the same way, the lecturer’s expectations for change in each of the 
three abilities. Previous research has shown that when the students’ expectations are the same as the 
lecturer’s, that is both students and lecturer are working towards the same degree of change (indicator of 
effectiveness), then students get high grades and both the lecturer and the students enjoy the course (criteria 
of effectiveness). The correlations between the in-course indicators and the post-course criteria are 
significant at p<=0.002, with n=56 (Bastick, 1995). 

Measuring effectiveness of teaching (ET) 

The 3AF involves more than using the feedback form and calculating the results. There are four steps 
in the application of the method. 

1. Pre-course peer justification of ratings 

In the design stage, before the course starts, the lecturer needs to use his/her professional expertise to 
decide on what should be the emphasis on the course for each of the three abilities. A rationale should 
justify this decision, and it needs to be peer agreed. The lecturer can then build his/her expectations of the 
three abilities into the teaching and the design of course assignments that give students the opportunities to 
demonstrate the required level of each ability. 

2. Explain and justify three abilities to students 

Near the beginning of the course the lecturer needs to make sure the students understand the three 
abilities and how they will be taught and assessed through the content of the course. 

3. Monitor ‘ as-is ’ and ‘should-be’ for students and lecturer 

The 3AF feedback form is completed by students and the lecturer when the lecturer is ready to monitor 
the course, or the administration is ready to assess the teaching. 

4. Calculating the effectiveness of teaching 

This is done in the following two stages (a) and (b). 

a) Calculate expected change for each student and lecturer: 

Change=( ‘ should-be ’ -‘ as-is ’ )/‘ as-is ’ 

b) Calculate alignment: 

Alignment=|Lecturer change-Student change| 

Zero is the perfect score 

The alignment is the Effective Teaching (ET) score. The ET score can be calculated for each student 
or as a mean for any group of students. Hence, the effectiveness of teaching can be monitored for any target 
group of interest - older students, students taking special electives, minority groups, gender balance, etc. 
Adjustments can be made to in-course teaching as necessary. 

Safeguards from variation in students’ set expectations 

Students’ expectations vary across Faculties and subjects, by years of education, previous experience 
and even by whether the course is compulsory or optional (Goldman, 1993). This creates an ‘unlevel 
playing field’ when traditional opinionnaires are used to assess faculty teaching. From the 3AF feedback 
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form the variation in students’ expectations can be calculated and the result used as a safeguard to protect 
the lecturer from inappropriate student expectations. It will be realized that all course evaluations depend 
on matching student expectations to the expectations of the course. Some aspects of the course can be 
presented in different ways to match different student expectations. For example, so called ‘learning styles’ 
can be matched by adopting different ‘teaching styles’. However, some aspects of the course may not be 
open to change to match student expectations, e.g. externally accredited content standards or the peer 
agreed emphases of the three abilities. Just as students expectations vary so does the flexibility of their 
expectations. If students’ expectations of these unchangeables cannot be altered to accept them, then 
teaching ratings will go down through no fault of the lecturer. Traditional opinionnaires penalize the lecturer 
because they make no allowance for large variations in unchangeable student expectations. However, the 
3AF allows the lecturer to show evidence that the original peer agreed emphases may not be appropriate 
for some groups on the course. This evidence can be used to either change the course expectations or 
change the student selection criteria. 



Institutional commitment to staff development 

The full Framework includes the commitment of the institution to develop faculty’s ability to use 
their subject specialism as a vehicle for explaining, teaching and assessing the three abilities. Institutional 
staff development support includes promoting academic freedom and professional responsibility, assisting 
faculty in making expectations explicit, in designing assessment opportunities for the three abilities and 
developing the ability of faculty to teach the three abilities using the content of their subject areas. As was 
mentioned at the start of this article, there is a saying in business that “what gets measured gets done and 
what gets rewarded gets repeated” (Friend, 1972). Quality teaching and quality learning get measured and 
get rewarded by the 3AF. 

Summary of how the 3AF and how it promotes a positive teaching and learning culture 

To use the 3AF in practice, lecturers explain to their students the three abilities and how they will be 
taught and assessed. When they wish to monitor the effectiveness of their teaching they ask the students to 
rate how they see the current emphasis of these three abilities and to rate how they would prefer the 
emphasis to be. The lecturer makes the same rating of the course. The indicator of effective teaching is that 
the students and the lecturer are working towards the same changes. This is measured by ‘the change 
expected by the students’ subtracted from ‘the change expected by the lecturer’. Zero is the perfect score 
on the total of the three abilities, and indicates perfect alignment. The alignment score is the measure of 
effective teaching and can be calculated for individual students, and the mean calculated for minority 
groups or for special comparisons e.g. to measure if the teaching more effective for males than for females. 



The method promotes a positive teaching and learning culture both directly and indirectly. It promotes 
a positive teaching and learning culture indirectly by encouraging forms of teaching and learning that 
faculty and students use to increase their valued assessment results, i.e. assessment driven teaching and 
learning. Namely, this method encourages teaching and learning that promotes students’ critical and 
evaluative thinking, high standards in technical skills and professional values because this is what is assessed 
in faculty teaching and in students learning. 

The assessment method also promotes a positive teaching and learning culture directly through student 
and faculty assessment support processes, as follows: 
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1. The institution promotes academic freedom and professional responsibility by confirming the 
lecturers’ professional prerogative to decide, and justify to their peers and their students, the emphasis 
they judge should be given to each of the three abilities on their courses. This is reinforced by 
recognising an assessment process that lecturers control. 

2. The institution promotes professional development by assisting faculty in making their professional 
expectations explicit in terms of the three abilities in their subject area and in assisting them to design 
coursework and examinations that offer opportunities for assessing these three abilities in their subjects. 

3. Faculty encourage students’ critical and evaluative thinking, to the extent faculty can justify this as 
desirable, by not assessing the correctness of professional competence, but by assessing the students’ 
justifications of why the novel aspects of their applications are appropriate. 

4. Faculty explicitly encourage professional attitudes, to the extent they can justify these as desirable, 
by assessing demonstrations of professional attitudes on course assignments. 



Generally, the development of technical skills is already well served by traditional methods of 

assessment. However, an interesting staff development programme would be to share methods of teaching 

professional competence and professional values. 
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